Is there a way to add it to all checks without to add this decorator here?

I don't see an easy was to do that as, since we don't use a lot of frameworks that the tracing client supports out of the box, we need to do this "manual instrumentation". I think having a single decorator per check isn't a bad trade off here though. I also don't think we necessarily want/need it for all checks right now 🙂

Yeah! That's fair. Thanks

I know import order matters for ddtrace currently, but we need to make this conditional too somehow.

Hmm, let me think about that one.

So it appears that the parent span has to be started in order for the underlying request to be patched and have an associated span be sent. Disabling the flag with the current state of the code submits 0 traces from the check.

Let's move this to the __init__.py of the base package, it should be early enough.

Let's move this to the __init__.py of the base package, it should be early enough.

This IMO should be:

tracer.trace('integration.check', service='IntegrationsTracing', resource=self.name)

This would allow us to have a main entry in the trace UI showing IntegrationsTracing, then the list of traces broken down by check.name. Each one would show (for now, or by default...) one traced function, integration.check.

My reasoning for making the Service name be the name of the class is so that we aren't locked into only decorating the check function.

If we make the service name static then the set of resources under that are going to be difficult to sift through if we add the decorator to a couple functions in a check.

Not sure how the two things are connected - you can have one big service and a bunch of resources grouped by it. Each resource would have one or more spans (likely one per function?) attached to it.

Let's not assume one trace == one function

let's remove this else

let's decorate this with @functools.wraps(func)

Let's use the inspect module to get any instance argument.

You can upgrade it to https://github.com/DataDog/dd-trace-py/releases/tag/v0.13.0 because 0.12.1 had some instability issues.

Done, thanks for the headsup

args[0].name is the name of the check, right? About the service name, you can use integrations-tracing just because usually we use that naming convention.

Yeah, args[0] is just the name of the check here, in this case it comes in the UI as openstack. I updated the service name to match that convention ^

when the Python interpreter is invoked from the Agent, do we know what integrations will be loaded? We may try to find a better place to patch enabled checks.

@masci Is this something you are familiar with? I can look into digging up the answer if not.

Do we know what the overhead of enabling this early on for all checks would be?

I changed this PR up a bit to put the patching of any libraries behind the datadog.yaml config flag. I think this is OK because its disabled by default, and only really expected to be enabled if an issue is being investigated so not for an extended period of time.

Let's put this in the if

from ..config import is_affirmative

Let's

if (
    cond1
    and cond2
):
    ...

Let's rename to trace, tracer, or add_tracing

I'm not sure we want it to name it the same thing as an exposed function from ddtrace? I chose trace_func at the time since its just a decorator that performs a standard manual instrumentation step for any function.

Perhaps tracing or add_tracing then?

I went with add_tracing

I didn't notice the name of the module before, sorry for the late comment but I think this should be tracing.py or something like that - decorator.py doesn't tell much about the role of this module in the package

t.get will raise if one of the args is not a dict-like object.

I think we need a better way to configure tracing, inspecting the args of the decorated function is not flexible and feels like looking for trouble.

My proposal

1. Move the settings from instance to init_config so tracing will be enabled per check and not per instance

2. Build a register tracking the checks that are configured to send traces and store it at the module-level so that it'll behave like a singleton at runtime. Also provide accessors so that a check instance can add itself to the registry.

__tracing_config = set()

def trace_check(check_object):
    __tracing_config.add(check_object)

def traced(wrapped, instance, args, kwargs):
    ...
    trace_check = instance in __tracing_config
    ...

the check itself should enable tracing in its __init__:

from datadog_checks.utils.decorator import trace_check

...

def __init__(self, name, init_config, agentConfig, instances=None):
    if init_config.get('trace_check', False):
        trace_check(self)